Detecting and analyzing performance anomalies of client-server based applications

ABSTRACT

An approach is provided for detecting and analyzing an anomaly in application performance in a client-server connection via a network. A status code of a response sent by a server to a client, a round trip latency time (RTT) of the response, and a time out of a connection between client and server are determined. Using a k-means clustering algorithm, buckets of RTT values clustered into lower and higher values, and running counts and means for the RTT values in each bucket, an RTT value is determined to exceed a threshold value. Based on the status code, the RTT value exceeding the threshold, and the connection time out, the anomaly is detected. Based on temporal and textual analyses of log entries and an environment analysis, candidate root causes of a failure that resulted in the anomaly are determined.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming priority to Ser.No. 14/869,129 filed Sep. 29, 2015, now U.S. Pat. No. 10,275,301, issuedApr. 30, 2019 the contents of which are hereby incorporated byreference.

BACKGROUND

The present invention relates to managing client-server basedapplication performance, and more particularly to detecting anddetermining root causes of mobile application faults and performancebottlenecks.

A mobile application has two main components in a client-server model:(1) a client side component running on the mobile device; and (2) aserver side component that responds to various requests from the client.Known techniques for detection and analysis of anomalies in mobileapplication performance utilize mobile analytics but provide either onlydevice analytics (i.e., by monitoring client side mobile applications)or only back-end analytics (i.e., by monitoring server sideinfrastructure), without taking into account details of client-serverinteractions end-to-end. For example, U.S. Patent ApplicationPublication No. 2010/0041391 discloses a client-focused mobile analyticsprocess that collects mobile device metrics at the mobile device. Knowntechniques for using the analytics to determine a root cause of theanomaly requires a significant amount of time for labor-intensive manualsearches to discover where the error originated. The manual searches arepainstaking because an application fault or a performance bottleneck mayoriginate in one place and time, but manifest itself at another placeand another time. Accordingly, there is a need for mobile analyticstechnique that has an integrated view across the device and the back-endand which is a less time-consuming technique for determining a likelyroot cause of the anomaly.

SUMMARY

In a first embodiment, the present invention provides a method ofdetecting and analyzing an anomaly in a performance of an application ina connection between client and server computers. The method includes afirst computer determining a time of a request from the client computerexecuting the application and an Internet Protocol (IP) address of theclient computer. The request is sent by the client computer to theserver computer via a communications network. The method furtherincludes based on the time of the request from the client computer andthe IP address of the client computer, the first computer selecting oneor more log entries from a plurality of log entries so that the selectedone or more log entries are relevant to the request. The method furtherincludes the first computer determining a status code of a response fromthe server computer, a round trip latency time (RTT) of the response,and an indication of whether the connection timed out. The response issent by the server computer to the client computer via the network andresponsive to the request. The method further includes based on thestatus code, the RTT, the indication of whether connection timed out, ora combination of the status code, the RTT, and the indication of whetherthe connection timed out, the first computer detecting the anomaly inthe performance of the application. The method further includes based ona temporal analysis and textual analysis of log entries associated withthe anomaly, and based on an environment analysis that determinesactivity of the client computer, the server computer, and the network,the first computer determining candidate root causes of a failure thatresulted in the anomaly. The failure is in the client computer, theserver computer, the network, or a combination of the client computer,the server computer, and the network.

In a second embodiment, the present invention provides a computerprogram product including a computer-readable storage device and acomputer-readable program code stored in the computer-readable storagedevice. The computer-readable program code includes instructions thatare executed by a central processing unit (CPU) of a computer system toimplement a method of detecting and analyzing an anomaly in aperformance of an application in a connection between client and servercomputers. The method includes the computer system determining a time ofa request from the client computer executing the application and anInternet Protocol (IP) address of the client computer. The request issent by the client computer to the server computer via a communicationsnetwork. The method further includes based on the time of the requestfrom the client computer and the IP address of the client computer, thecomputer system selecting one or more log entries from a plurality oflog entries so that the selected one or more log entries are relevant tothe request. The method further includes the computer system determininga status code of a response from the server computer, a round triplatency time (RTT) of the response, and an indication of whether theconnection timed out. The response is sent by the server computer to theclient computer via the network and responsive to the request. Themethod further includes based on the status code, the RTT, theindication of whether connection timed out, or a combination of thestatus code, the RTT, and the indication of whether the connection timedout, the computer system detecting the anomaly in the performance of theapplication. The method further includes based on a temporal analysisand textual analysis of log entries associated with the anomaly, andbased on an environment analysis that determines activity of the clientcomputer, the server computer, and the network, the computer systemdetermining candidate root causes of a failure that resulted in theanomaly. The failure is in the client computer, the server computer, thenetwork, or a combination of the client computer, the server computer,and the network.

In a third embodiment, the present invention provides a computer systemincluding a central processing unit (CPU); a memory coupled to the CPU;and a computer-readable storage device coupled to the CPU. The storagedevice includes instructions that are executed by the CPU via the memoryto implement a method of detecting and analyzing an anomaly in aperformance of an application in a connection between client and servercomputers. The method includes the computer system determining a time ofa request from the client computer executing the application and anInternet Protocol (IP) address of the client computer. The request issent by the client computer to the server computer via a communicationsnetwork. The method further includes based on the time of the requestfrom the client computer and the IP address of the client computer, thecomputer system selecting one or more log entries from a plurality oflog entries so that the selected one or more log entries are relevant tothe request. The method further includes the computer system determininga status code of a response from the server computer, a round triplatency time (RTT) of the response, and an indication of whether theconnection timed out. The response is sent by the server computer to theclient computer via the network and responsive to the request. Themethod further includes based on the status code, the RTT, theindication of whether connection timed out, or a combination of thestatus code, the RTT, and the indication of whether the connection timedout, the computer system detecting the anomaly in the performance of theapplication. The method further includes based on a temporal analysisand textual analysis of log entries associated with the anomaly, andbased on an environment analysis that determines activity of the clientcomputer, the server computer, and the network, the computer systemdetermining candidate root causes of a failure that resulted in theanomaly. The failure is in the client computer, the server computer, thenetwork, or a combination of the client computer, the server computer,and the network.

Embodiments of the present invention provides a general solution fordiagnostics and forensics of distributed applications by collectingrelevant information from all application components, accuratelycorrelating client and server activities, classifying faults andbottlenecks, and identifying sources and underlying causes of the faultsand bottlenecks at runtime. The automated generation of diagnostic cluesor determination of root causes significantly reduces administrativelabor time as well as system downtime. A learning module learns frompast behavior and user feedback about whether proposed anomalies areactual anomalies, which improves future identification of anomalies byreducing false positive and false negative rates for anomalydetermination.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for detecting and analyzing aperformance anomaly of a client-server based application, in accordancewith embodiments of the present invention.

FIG. 2 is a flowchart of a process of detecting and analyzing aperformance anomaly of a client-server based application, where theprocess is implemented in the system of FIG. 1, in accordance withembodiments of the present invention.

FIG. 3 is a flowchart of a process of determining candidate root causesof a performance anomaly detected by the process of FIG. 2, inaccordance with embodiments of the present invention.

FIG. 4 is a flowchart of a process of refining a detection ofperformance anomalies, where the detection had resulted from the processof FIG. 2, in accordance with embodiments of the present invention.

FIGS. 5A-5B are examples of detecting performance anomalies based onround trip latency times, as utilized in the process of FIG. 2, inaccordance with embodiments of the present invention.

FIG. 6A is an example of a structure containing a specification ofHypertext Transfer Protocol (HTTP) client-server connection parametersutilized in the process of FIG. 2, in accordance with embodiments of thepresent invention.

FIG. 6B is an example of a structure containing a specification ofclient device environment parameters utilized in the process of FIG. 2,in accordance with embodiments of the present invention.

FIG. 6C is an example of a structure containing a specification ofapplication server log entry parameters utilized in the process of FIG.2, in accordance with embodiments of the present invention.

FIG. 7A is an example of a user interface presenting a list of faultsand anomalies detected in the process of FIG. 2, in accordance withembodiments of the present invention.

FIG. 7B is an example of a user interface that includes candidate rootcauses determined in the process of FIG. 3, in accordance withembodiments of the present invention.

FIG. 8 is a block diagram of a computer that is included in the systemof FIG. 1 and that implements the processes of FIG. 2, FIG. 3, and FIG.4, in accordance with embodiments of the present invention.

DETAILED DESCRIPTION

Overview

Embodiments of the present invention detect faults and performanceanomalies in client-server based mobile applications. The detection offaults and performance anomalies include generating a taxonomy of faultsand performance issues occurring in client-server mobile applications,tracking and collecting distributed data. Embodiments of the presentinvention utilize statistical algorithms applied to the collected datato evaluate performance, determine anomalous behavior that results inpoor performance, and determine likely root causes of the faults andpoor performance. In one embodiment, the system for anomaly detection isable to learn from verifications of root causes of faults and unexpectedpoor performance and then use that learning to refine the system,thereby improving the adaptability of the system. A graphical userinterface (GUI) may present analysis results and additional diagnosticclues to users.

A client-server based mobile application includes client side and serverside components. The client side component is a software applicationexecuted on a mobile device (e.g., a smartphone). The server sidecomponent responds to requests sent from the client side component.Instead of providing only mobile device analytics or only back-endanalytics, embodiments of the present invention advantageously providemobile analytics that provides an integrated view of details ofclient-server interactions end-to-end (i.e., across the mobile deviceand the back-end).

As used herein, an application is dependent upon client-serverinteractions working properly. A location of a fault in the applicationmay be in a client side component, a server side component, or in thenetwork used by the client-server interactions. As used herein, a faultor failure is a fault in the application that manifests itself duringthe course of a client-server interaction via a client-server connectionin the context of an application session. The client-server connectionmay utilize any application layer protocol (e.g., Hypertext TransferProtocol (HTTP)). Client side or server side components may fail inunanticipated ways, thereby negatively affecting client-serverinteractions. Failures that negatively affect client-server interactionsmay include, for example, logic failures (i.e., bugs in software), aclient or server application being starved of resources on the computerthat executes the application, or a network link, router, or switchexperiencing an outage which results in a client-server disconnection.Other examples of faults may include an application crash resulting froma mobile device running out of memory, a server application hitting anexception (e.g., HTTP Connections returns a 500 error code (InternalServer Error)), or connections time out after a pre-specified timelimit.

Although embodiments presented herein focus on faults and performanceanomalies in client-server based mobile applications, it is apparent tothose skilled in the art that the embodiments may be extended to otherclient-server based applications that are executed on computers that arenot mobile devices.

System for Detecting and Analyzing a Performance Anomaly of aClient-Server Based Mobile Application

FIG. 1 is a block diagram of a system 100 for detecting and analyzing aperformance anomaly of a client-server based application, in accordancewith embodiments of the present invention. System 100 includes acomputer 102, a computer mobile application stack 104, a mobile device106, and a network 108. Mobile device 106 executes a software-basedmobile application 109.

Although FIG. 1 includes mobile application 109 executing on mobiledevice 106, other embodiments substitute another computer in place ofmobile device 106, where the other computer is not a mobile device andexecutes another software application in place of mobile application109.

Mobile application 109 is a client-server based application, wheremobile device 106 is the client, which communicates via network 108 witha server included in customer mobile application stack 104. In otherembodiments, system 100 includes one or more other mobile devices (notshown) executing mobile application 109 or other mobile application(s)in application session(s) with customer mobile application stack 104 vianetwork 108. Mobile application 109 includes a software-based deviceagent 110 and a logging framework 112. Device agent 110 accesses logs inlogging framework 112 to collect data from the logs about an applicationsession in which mobile application 109 is participating. The collecteddata includes the time of client requests from mobile application 109and the IP address of the mobile device 106. In one embodiment, deviceagent 110 runs as a background service on mobile device 106.

Customer mobile application stack 104 includes a web server 114, anapplication server 116, and a database server 118, which areparticipating in the aforementioned application session. Customer mobileapplication stack 104 also includes a software-based back-endinfrastructure agent 120, which collects data about the applicationsession from logs provided by web server 114, application server 116,and database server 118. Alternatively, customer mobile applicationstack may include back-end infrastructure agent 120 and exactly oneserver or other numbers of servers not shown in FIG. 1, where theserver(s) provide one or more logs from which back-end infrastructureagent 120 collects data about the application session.

Network 108 includes the following components: one or more routers (notshown), one or more switches (not shown), and one or more firewalls (notshown). Network 108 also includes a software-based networkinfrastructure agent 122, which collects data about the networkconnection being used by the application session, where the data iscollected from log(s) provided by one or more of the aforementionedcomponents included in network 108.

Computer 102 includes software-based tools that monitor and analyzeparticular components of system 100 in isolation: a mobile userexperience device monitoring and analytics tool 124, a mobileinfrastructure server monitoring and analytics tool 126, and a networkinfrastructure monitoring and analytics tool 128.

Mobile user experience device monitoring and analytics tool 124 monitorsmobile device 106, collects activity and health information datareceived from device agent 110, and performs application usage analyticsfor mobile application 109. The activity and health informationindicates how and when a user uses mobile application 109 and includesthe time of the client request from mobile application 109 and the IPaddress of the mobile device 106 which is the source of the request.Mobile user experience device monitoring and analytics tool 124 receivesfrom device agent 110: (1) a response code, which is a status code ofthe response from web server 114, and (2) a round-trip latency time ofthe response to the request from mobile application 109. Mobile userexperience device monitoring and analytics tool 124 also computesaggregate statistics across multiple users and multiple devices (notshown) which have respective application sessions with customer mobileapplication stack 104 via network 108.

Mobile infrastructure server monitoring and analytics tool 126 performsserver-side analytics by monitoring and analyzing server logs and serverhealth parameters, which are received from back-end infrastructure agent120. The server logs are logs provided by web server 114, applicationserver 116, and database server 118. An entry in a server log isfree-form text written by an application developer to trace anapplication run. Server log entries are textually analyzed using datamining or pattern matching techniques. Mobile infrastructure servermonitoring and analytics tool 126 also generates output in the form ofstatistics and tables that are presented to the user through a GUI.

Network infrastructure monitoring and analytics tool 128 monitors thestate and activity logs of the communication connection and componentsof network 108 by receiving information about the state and activitylogs from network infrastructure agent 122. The aforementionedcommunication connection is the connection being used by mobileapplication 109 to communicate with one or more of the server componentsof customer mobile application stack 104. The activity logs are logsprovided by components (not shown), such as routers and switches,included in network 108.

Mobile user experience monitoring and analytics tool 124, mobileinfrastructure server monitoring and analytics tool 126, and networkinfrastructure monitoring and analytics tool 128 sends theirrespectively received activity and health information data to asoftware-based client-server-network activity correlation,infrastructure and user experience analytics, diagnostics, and forensicsengine 130 (hereinafter, simply “diagnostics and forensics engine”).Diagnostics and forensics engine 130 utilizes the data from tools 124,126, and 128 to detect anomalies in server responses to requests frommobile application 109 and to determine candidates for root causes offailures in system 100 that resulted in the anomalies. Diagnostics andforensics engine 130 sends information about the anomalies and candidateroot causes to an alert notification generation module 132, whichgenerates an alert about the anomalies and candidate root causes to beviewed and annotated by a user.

A root cause of a failure in system 100 may be located in (1) clientside components (e.g., mobile application 109 is running slowly ormobile device 106 is overloaded due to high CPU usage or the amount ofmemory remaining is low), (2) server side components (e.g., applicationserver 116 is operating slowly or is overloaded due to high CPU usage,low memory, or high disk input/output (I/O) activity), or (3)communication network components (e.g., network speed is low, or thereis an outage or overload of network components including routers andswitches). High RTT values can have multiple simultaneous root causes(e.g., an overloaded database managed by database server 118 and arouter outage).

A learning and rule updating module 134 receives feedback from the userreviewing and annotating the alerts, uses the feedback to update rulesthat determine anomalies based on log entries from mobile device 106,network 108, and server components of customer mobile application stack104, and use the updated rules to refine the anomaly determination forsubsequent requests generated by mobile application 109 and sent to oneof the server components of the customer mobile application stack 104.

Alert notification generation module 132 sends the alerts about theanomalies and candidate root causes to a GUI dashboard and/or a report136, which is viewed by a user of computer 102. Diagnostics andforensics engine 130, mobile user experience monitoring tool 124, andmobile infrastructure server monitoring and analytics tool 126 sendmonitored data from the client and server, results of the analysis ofthe monitored data, and statistics and tables to dashboard and/or report136. The GUI dashboard 136 provides detailed information about anomalousevents as a result of an alert or in response to user queries or entriesin a search interface.

The functionality of the components shown in FIG. 1 is described in moredetail in the discussion of FIG. 2, FIG. 3, FIG. 4, and FIG. 5 presentedbelow.

Process for Detecting and Analyzing a Performance Anomaly of aClient-Server Based Mobile Application

FIG. 2 is a flowchart of a process of detecting and analyzing aperformance anomaly of a client-server based application, where theprocess is implemented in the system of FIG. 1, in accordance withembodiments of the present invention. The process of FIG. 2 starts atstep 200. An application session is ongoing between a client (i.e.,mobile device 106 (see FIG. 1)) and a server (i.e., web server 114,application server 116, or database server 118 in FIG. 1). In step 202,diagnostics and forensics engine 130 (see FIG. 1) determines a time of aclient request from mobile application 109 (see FIG. 1) and the IPaddress of the mobile device 106 (see FIG. 1), which is the source ofthe client request.

In step 204, based on the time of the client request and the IP addressof the client determined in step 202, diagnostics and forensics engine130 (see FIG. 1) selects one or more relevant log entries from logsprovided by logging framework 112 (see FIG. 1), the server, andcomponents of network 108 (see FIG. 1). The relevant log entries includeinformation monitored by device agent 110 (see FIG. 1), back-endinfrastructure agent 120 (see FIG. 1), and network infrastructure agent122 (see FIG. 1). Device agent 110 (see FIG. 1) monitors activity of theapplication session by monitoring application logs, performingmethod-level tracking, and obtaining network connection and sessioninformation. Back-end infrastructure agent 120 (see FIG. 1) monitorsactivity of the application session by monitoring application server anddatabase server logs, and obtaining network connection and sessioninformation. Network infrastructure agent 122 (see FIG. 1) monitorsactivity of the application session by monitoring network router logsand network switch logs, if such logs are available.

Prior to step 206, device agent 110 (see FIG. 1) records or calculatesthe following information: (1) a response code in the server response tothe client request, (2) a round-trip latency time (RTT) of the responseto the client request, and (3) an indication of whether the connectionbetween the client and the server timed out, and subsequently, deviceagent 110 (see FIG. 1) sends the aforementioned recorded or calculatedinformation to mobile user experience device monitoring and analyticstool 124 (see FIG. 1). In step 206, based on receiving theaforementioned recorded or calculated information from mobile userexperience device monitoring and analytics tool 124 (see FIG. 1),diagnostics and forensics engine 130 (see FIG. 1) determines theresponse code in the server response to the client request, the RTT ofthe response to the client request, and the indication of whether theconnection between the client and the server timed out. The serverresponse is the response sent by the server participating in theapplication session, responsive to the client request being received bythe server. The RTT is the amount of time from the time at which mobileapplication 109 (see FIG. 1) sends a request to the server to the timeat which the mobile application 109 (see FIG. 1) receives a responsefrom the server. A connection times out in response to the clientsending a request to the server and the server not responding to therequest within a predetermined time period. The response code is amessage included in a response a server sends to a client in response tothe client sending a request to the server. The response code indicateswhether the server performed the function requested by the client or wasunable to perform the function. In one embodiment, the response code isa HTTP status code, where a response code of 200 indicates that theserver properly performed its function in response to a request from theclient, a response code of 400 through 499 indicates that the clientsent a malformed request, and therefore the server was unable to fulfillthe request, and a response code of 500 through 599 is a failure codethat indicates the server did not properly perform its function inresponse to the request from the client.

In step 208, based on the response code, the RTT, or the indication ofwhether the connection between the client and server timed out, or acombination of the response code, the RTT, and the indication of whetherthe connection timed out, diagnostics and forensics engine 130 (seeFIG. 1) detects an anomaly in the server response to the client request.Diagnostics and forensics engine 130 (see FIG. 1) marks response codesthat indicate the server was unable to properly perform its function asindicating performance anomalies and marks connections that timed out asanomalous client-server interactions.

Diagnostics and forensics engine 130 (see FIG. 1) utilizes one or moreknown statistical methods to determine how large a RTT value must be tobe considered an indication of an anomaly. In one embodiment,diagnostics and forensics engine 130 (see FIG. 1) utilizes a k-meansclustering algorithm to determine a threshold RTT value (i.e.,threshold) above which diagnostics and forensics engine 130 (see FIG. 1)marks RTT values as anomalous. Using the k-means clustering algorithm,where k=2, diagnostics and forensics engine 130 (see FIG. 1) partitionsthe gathered RTT values into two sets: a lower values cluster (i.e., C₁)and a higher values cluster (i.e., C₂). Diagnostics and forensics engine130 (see FIG. 1) determines a mean μ₁ and a standard deviation σ₁ ofcluster C₁ and a mean μ₁ and a standard deviation σ₂ of cluster C₂. Ifclusters C₁ and C₂ overlap to a high extent (i.e., exceeding apredetermined amount of overlap), then diagnostics and forensics engine130 (see FIG. 1) chooses anomalies only from the higher values cluster.Two standard deviations from the mean is designated as sufficientlyanomalous. That is, if μ₁+σ₁≥μ₂ then diagnostics and forensics engine130 (see FIG. 1) determines that threshold=μ₂+2σ₂, elsethreshold=μ₁+2σ₁. Again, the aforementioned threshold computationalgorithm is only one embodiment; other embodiments may utilize otheralgorithms to partition the RTT values and compute the threshold.

Diagnostics and forensics engine 130 (see FIG. 1) flags a performanceanomaly if the RTT value of a given client-server connection exceeds thecomputed threshold.

If a user has enough domain knowledge to know what a high RTT value is,that user can manually set the value of threshold. A minimum or maximumvalue of threshold can be pre-set depending on the type of theapplication 109 (see FIG. 1).

For example, diagnostics and forensics engine 130 (see FIG. 1)determines that a normal RTT for requests is approximately 30milliseconds and detects that a particular request has a RTT of fiveseconds. Diagnostics and forensics engine 130 (see FIG. 1) detects thatthe five second RTT exceeds the threshold amount, which indicates ananomaly. The anomaly indicates a performance bottleneck or otherperformance issue, or may indicate a component failure.

In one embodiment, diagnostics and forensics engine 130 (see FIG. 1)continuously tracks RTT values that result from streaming data frommultiple mobile devices having application sessions in system 100 (seeFIG. 1). Because it is infeasible to process the entire set ofhistorical RTT values every time new RTT values are available todetermine the aforementioned clusters, diagnostics and forensics engine130 (see FIG. 1) utilizes a streaming threshold computation algorithm,which is a variation of the threshold computation algorithm describedabove.

In the streaming threshold computation algorithm, diagnostics andforensics engine 130 (see FIG. 1) performs the following steps afterusing the k-means clustering algorithm to compute an initial value ofthreshold:

1. Divide the space of RTT values into fixed size buckets (e.g., 0-50milliseconds, 50-100 milliseconds, 100-150 milliseconds, etc.).

2. Maintain running counts and means for RTT values in each bucket.

3. Maintain boundary value to determine which buckets fall in the lowervalue cluster and which fall in the higher value cluster (e.g., maintaina boundary value of 300 milliseconds).

4. For every new batch of new RTT values, (i) determine the bucket thateach new RTT value falls in, (ii) assign each new RTT value to anappropriate bucket, (iii) re-compute counts and means for each bucket,(iv) re-balance the clusters to ensure that values in both clusters arecloser to their respective cluster means, and (v) move buckets on theboundary up to the higher value cluster or down to the lower valuecluster monotonically until further movement is not possible.

5. Re-compute the mean and standard deviation for each cluster.

6. Re-compute threshold using the k-means clustering algorithm describedabove.

7. If any of the new RTT values exceeds threshold, flag those RTTvalue(s) and the associated client-server connections in the new batchas anomalous.

In other embodiments, diagnostics and forensics engine 130 (see FIG. 1)may utilize a variation of the streaming threshold computationalgorithm, which may employ different bucket sizes and counts, adifferent threshold computation formula, and/or variable sizes for everynew batch of RTT values.

In step 210, diagnostics and forensics engine 130 (see FIG. 1) performstemporal analysis and then textual analysis to filter log entries whichare relevant to the anomaly detected in step 208. The log entriesresulting from the filtering in step 210 are hereinafter also referredto as the filtered log entries.

In step 212, diagnostics and forensics engine 130 (see FIG. 1) obtainsinformation from device agent 110 (see FIG. 1), back-end infrastructureagent 120 (see FIG. 1), and network infrastructure agent 122 (seeFIG. 1) to perform an environment analysis, which determines health andquality of service (QoS) indicators (i.e., environment parameters) ofthe server, the client, and network 108 (see FIG. 1) in the applicationsession. In one embodiment, the health and QoS indicators indicatewhether the server has adequate unused memory and whether the CPU usageof the server is spiking in excess of a predetermined amount.

In step 212, the information obtained from device agent 110 includesindicators of the health of client side components, including indicatorsof CPU usage, memory usage, and I/O activity in mobile device 106 (seeFIG. 1), the information from back-end infrastructure agent 120 (seeFIG. 1) includes indicators of CPU usage, memory usage, and I/O activityin servers in customer mobile application stack 104 (see FIG. 1), andthe information from network infrastructure agent 122 (see FIG. 1)includes QoS parameters such as bandwidth, latency, and jitter.

In step 214, based on the filtered log entries and based on theenvironment analysis performed in step 212, diagnostics and forensicsengine 130 (see FIG. 1) determines one or more candidates of the rootcause(s) of a failure in system 100 (see FIG. 1) that resulted in theanomaly detected in step 208. Hereinafter, the one or more candidates ofthe root cause(s) are referred to as the candidate root cause(s).

In step 216, alert notification generation module 132 (see FIG. 1)generates and presents an alert, which includes the candidate rootcause(s) along with (1) one or more of the filtered log entries thatspecify attributes of the candidate root cause(s) and (2) arepresentation (e.g., statistics, table, or diagram) of the health andQoS indicators that specify attributes of the candidate root cause(s).In one embodiment, diagnostics and forensics engine 130 (see FIG. 1)determines a type of the alert and sends the alert to users who haveroles that are relevant to the type of the alert. Alert notificationgeneration module 132 (see FIG. 1) may send the alert to the users overspecified channels such as emails, push notifications, and text (i.e.,Short Message Service (SMS)) messages. In one embodiment, alertnotification generation module 132 (see FIG. 1) presents the alert viaGUI dashboard and/or report 136 (see FIG. 1).

The process of FIG. 2 ends at step 218.

FIG. 3 is a flowchart of a process of determining candidate root causesof an anomaly detected by the process of FIG. 2, in accordance withembodiments of the present invention. The process of FIG. 3 expands thesteps of 210, 212, 214, and 216 in FIG. 2 and starts at step 300.

In step 302, the anomaly detected in step 208 (see FIG. 2) is input intothe process of FIG. 3. In step 304, the environment parameters resultingfrom the environment analysis performed in step 212 (see FIG. 2) areinput into the process of FIG. 3.

In step 306, diagnostics and forensics engine 130 (see FIG. 1)determines a time window (i.e., period of time) that is likely toinclude the time at which a root cause caused a failure of system 100(see FIG. 1), which caused the anomaly input in step 302.

In step 308, based on the time window determined in step 306,diagnostics and forensics engine 130 (see FIG. 1) selects relevantcomponents from among the client (i.e., mobile device 106 in FIG. 1),servers, and network 108 (see FIG. 1).

In step 310, diagnostics and forensics engine 130 (see FIG. 1) performstemporal analysis by selecting relevant log entries from logs providedby the relevant client, server, and network components selected in step308, where the selection of the log entries is based on an approximatetime window of fault (i.e., select only log entries whose timestamps arewithin the approximate time window of fault). In one embodiment,diagnostics and forensics engine 130 (see FIG. 1) determines that afaulty connection indicated by a high RTT started at time T₁ and endedat time T₂, thereby indicating a high likelihood that the fault occurredbetween time T₁ and time T₂. Because there is a decreasing likelihoodthat the original fault occurred before time T₁, diagnostics andforensics engine 130 (see FIG. 1) generates the approximate time windowof fault as T₁−w to T₂ which extends the window of time T₁ to time T₂ toinclude a predefined amount of time w before time T₁.

In step 312, diagnostics and forensics engine 130 (see FIG. 1) performstextual analysis by filtering the log entries selected in step 310 basedon known and learned keywords. As used herein, a keyword is defined as aword or phrase that is predetermined to be an indicator of an anomaly ofsystem 100 (see FIG. 1). In one embodiment, steps 310 and 312 areincluded in step 210 in FIG. 2.

In one embodiment, the textual analysis in step 312 includes extractingkeywords from connection information (e.g., from the URL or from themessage payload) and utilizing a database of relevant keywords (e.g.,words including “exception,” “waiting,” “password,” “failure,” etc.).Diagnostics and forensics engine 130 (see FIG. 1) attempts to matchwords or phrases in the log entries to the database of keywords. Logentries that have words or phrases that match entries in the database ofkeywords are candidates for determining causes of anomalies.

In step 314, which follows step 304, step 308 and step 312, diagnosticsand forensics engine 130 (see FIG. 1) determines health (e.g., CPU usageand memory usage) and QoS (e.g., input/output activity) indicators forthe time window determined in step 306. Also in step 314, and based onthe health and QoS indicators, diagnostics and forensics engine 130 (seeFIG. 1) determines the activity of the relevant client, server, andnetwork components selected in step 308.

In step 316, diagnostics and forensics engine 130 (see FIG. 1) generatesan activity map, which (1) indicates whether each of the client, server,and network components is active or inactive in the context of theanomaly input in step 302; and (2) indicates whether active client,server, or network components were (i) performing tasks relevant tomobile application 109 (see FIG. 1) or (ii) busy performing extraneouswork. In one embodiment, step 316 is included in step 212 in FIG. 2. Viathe activity map, diagnostics and forensics engine 130 (see FIG. 1)classifies each anomaly as having a root cause whose location is (1) themobile device 106 (see FIG. 1), (2) server components, or (3) componentsof the network channel through which the client and server communicate.

After step 316 and prior to step 318, diagnostics and forensics engine130 (see FIG. 1) determines the likely location of the root cause of thefailure that resulted in the anomaly input in step 302 by performing thefollowing steps:

1. Determine a subset of the log entries, where the entries in thesubset correspond to mobile application 109 (see FIG. 1). The subsequentsteps in determining the likely location of the root cause are performedonly on the subset of log entries. Diagnostics and forensics engine 130(see FIG. 1) determines the log entries having a correspondence tomobile application 109 (see FIG. 1) by utilizing application identifiers(IDs) or thread IDs included in annotations in the log entries.

2. Perform an inactivity based determination of the source of the faultby checking for an absence of log entries in the approximate time windowof fault, which indicates a high probability that system 100 (seeFIG. 1) is overloaded and is therefore not able to devote resources tomobile application 109 (see FIG. 1). Module(s) exhibiting the inactivityare likely to be sources of performance-related faults. For example, ifa back-end application server instance logged very little or noinformation, whereas the client device and network components (e.g.,router) exhibited significant logging activity, then the applicationserver is the likely source of the fault.

3. Perform an overload-based (i.e., heavy logging activity based)determination of the source of the fault by checking for a substantialnumber of log entries in the approximate time window of fault (i.e., thenumber of log entries exceeds a predetermined threshold amount), but fewor none of these log entries are relevant to mobile application 109 (seeFIG. 1). If the aforementioned heavy logging activity is detected, itindicates that the server is overloaded and the mobile application 109(see FIG. 1) is starved of CPU cycles to run, thereby adding to thenetwork connection delay. The detection of the aforementioned overloadindicates that the server is the likely source of the fault.

4. Correlate logging activity with connection duration. For example, if(1) the client logs indicate activity unrelated to mobile application109 (see FIG. 1), (2) the server logs indicate that the server-side ofthe application is running, and (3) network logs indicate that thenetwork is not undergoing a delay, then an overloaded client device(i.e., mobile device 106 in FIG. 1) is the likely source of the fault.

Diagnostics and forensics engine 130 (see FIG. 1) also utilizes healthindicators determined in step 314 to yield diagnostic information. Forexample, if diagnostics and forensics engine 130 (see FIG. 1) detectshigh I/O activity or high memory usage on the machine running theapplication server during the approximate time window of fault, then itis likely that the machine was responsible for the fault. Diagnosticsand forensics engine 130 (see FIG. 1) generates graphs and statistics toindicate the variation of CPU usage, memory usage, battery usage, andI/O activity during the approximate time window of fault. At a giventime, more than one of the client, server, and network could beunderperforming and causing performance anomalies, and in such a case,embodiments of the present invention may present inconclusive results.In the case of inconclusive results, diagnostics and forensics engine130 (see FIG. 1) presents to a user log entries, charts, and statisticsfor the approximate time window of fault, so that the user can thenmanually inspect the results to determine a likely root cause.

In step 318, based on the activity map generated instep 316, diagnosticsand forensics engine 130 (see FIG. 1) determines the candidate rootcause(s) of the failure that resulted in the anomaly input in step 302.The process in FIG. 3 is a best effort procedure and therefore there isnot a guarantee that the precise root cause will be determined. Ifmultiple candidate root causes are determined in step 318, then step 318also includes diagnostics and forensics engine 130 (see FIG. 1)determining a confidence in each candidate root cause being the actualroot cause. In one embodiment, step 318 is included in step 214 in FIG.2.

In step 320, diagnostics and forensics engine 130 (see FIG. 1) generatesa display of the candidate root cause(s), along with evidence supportingthe determination of the candidate root cause(s) for viewing by a userof computer 102 (see FIG. 1). If there are multiple root causesdetermined in step 318, then step 320 includes diagnostics and forensicsengine 130 (see FIG. 1) displaying the candidate root causes in theorder of the confidence determined in step 318. In one embodiment, theorder of confidence is based on the number of log entries selected andfiltered out through the temporal and textual analyses in steps 310 and312 as indicating an anomaly and the number of health and QoS indicatorsthat indicate an anomaly as a result of the environment analysis in step314. For example, if the temporal and textual analyses yield 10 logentries on server logs that indicate faults but yield zero similarentries on client logs, and if the environment analysis indicates a CPUspike on the client side, but no anomalies in the health or QoSindicators of the server, then compared to the client, the server isassigned a higher confidence of being the location of the root causebecause the 10 log entries plus zero health and QoS indicators of theserver is greater than the zero log entries plus one health and QoSindicator for the client.

The display of the candidate root causes in step 320 advantageouslyfilters out irrelevant and extraneous information, which allows a userto focus on an amount of data (i.e., candidate root causes) that issubstantially smaller than the information provided by known diagnostictechniques, thereby leading to a quicker manual analysis of thecandidate root causes to determine an actual root cause of the anomaly.

The process of FIG. 3 ends at step 322.

FIG. 4 is a flowchart of a process of refining a detection of anomalies,where the detection had resulted from the process of FIG. 2, inaccordance with embodiments of the present invention. The process ofFIG. 4 starts at step 400. In step 402, diagnostics and forensics engine130 (see FIG. 1) collects attributes of the anomaly detected in step 208(see FIG. 2) and sends the attributes to a machine learning processperformed by learning and rule updating module 134 (see FIG. 1). Thecollected attributes include RTT, an indication of whether theconnections timed out, delay value of the connections, server details,application details, the set of functions that are executing or areplanned to be executed, service uniform resource locator (URL) that isbeing called, etc.

In step 404, based on a role of a user, a type of an alert, and a storedassociation between the role and the type of an alert, alertnotification generation module 132 (see FIG. 1) sends the alert to theuser via GUI dashboard and/or report 136 (see FIG. 1), where the alertdescribes the anomaly detected in step 208 (see FIG. 2).

In step 406, diagnostics and forensics engine 130 (see FIG. 1) receivesfrom the user feedback or an annotation of the alert, which specifiesthe anomaly described in the alert as being true or false (i.e.,accurately identified as an anomaly or inaccurately identified as ananomaly).

In step 408, learning and rule updating module 134 (see FIG. 1) receivesand utilizes the true or false specification in the feedback orannotation received in step 406 as the label of the machine learningprocess. By incorporating the true or false specification into themachine learning process, false positives are detected and eliminated insubsequent anomalies detected by system 100 (see FIG. 1), therebyimproving the accuracy of anomaly determination by system 100 (see FIG.1).

In step 410, learning and rule updating module 134 (see FIG. 1)generates a machine learning model based on the attributes of theanomaly collected in step 402. The machine learning model includes rulesfor determining whether an event is an anomaly in system 100 (see FIG.1).

In step 412, diagnostics and forensics engine 130 (see FIG. 1)determines a next anomaly in system 100 (see FIG. 1) based on rules inthe machine learning model generated in step 410. The accuracy of thedetermination of the next anomaly is improved from the determination ofprior anomalies because of the rules in the machine learning model.

In step 414, learning and rule updating module 134 (see FIG. 1) updatesthe machine learning model continuously or at a specified periodicity.

In step 416, diagnostics and forensics engine 130 (see FIG. 1)determines subsequent anomalies based on the machine learning modelupdated in step 414, thereby further improving the anomaly detectionaccuracy.

The process of FIG. 4 ends at step 418.

EXAMPLES

FIGS. 5A-5B are examples of identifying performance anomalies based onround trip latency times, in accordance with embodiments of the presentinvention. Diagnostics and forensics engine 130 (see FIG. 1) determinesRTT values in step 206 (see FIG. 2) which are data points placed in ascatter plot depicted in FIG. 5A. Diagnostics and forensics engine 130(see FIG. 1) determines that the data points are clustered into a firstgroup 502 of RTT data points and a second group 504 of RTT data points.Diagnostics and forensics engine 130 (see FIG. 1) determines that theRTT data points in first group 502 are clustered in a narrow band inwhich the RTT values are low enough to ensure a good user experience forthe user of mobile application 109 (see FIG. 1) (i.e., the data pointsin first group 502 indicate normal RTT values). Diagnostics andforensics engine 130 (see FIG. 1) in step 208 (see FIG. 2) detectsperformance anomalies at the times associated with the RTT data pointsin second group 504 by determining that the RTT data points in secondgroup 504 are not in the aforementioned narrow band of normal RTTvalues, and are high enough to ensure a negative user experience for theuser of mobile application 109 (see FIG. 1). Diagnostics and forensicsengine 130 (see FIG. 1) filters out the RTT data points in second group504 and for each of the performance anomalies, determines whatcorresponding part of system 100 (see FIG. 1) is the origin of theanomaly. In one embodiment, diagnostics and forensics engine 130 (seeFIG. 1) makes no a priori assumptions about what RTT values are normal.

The frequency of RTT values determined in step 206 (see FIG. 2) may beplaced in a frequency graph depicted in FIG. 5B. Diagnostics andforensics engine 130 (see FIG. 1) determines that the RTT frequenciesare clustered into a first group 552 of RTT frequencies and a secondgroup 554 of RTT frequencies. Diagnostics and forensics engine 130 (seeFIG. 1) determines that the RTT frequencies in first group 552 arehigher than the RTT frequencies in second group 554, and thereforedetermine that the RTT values associated with first group 552 offrequencies are normal RTT values and the RTT values associated withsecond group 554 of frequencies indicate performance anomalies. That is,the RTT values that are relatively high due to faults in system 100 (seeFIG. 1) tend to occur less frequently.

FIG. 6A is an example of a structure 600 containing a specification ofHypertext Transfer Protocol (HTTP) client-server connection parametersutilized in the process of FIG. 2, in accordance with embodiments of thepresent invention. In step 204 (see FIG. 2), the selected log entriesmay include HTTP client-server connection parameters which are specifiedby structure 600.

FIG. 6B is an example of a structure 620 containing a specification ofclient device environment parameters utilized in the process of FIG. 2,in accordance with embodiments of the present invention. In step 212(see FIG. 2), the environment analysis may utilize environmentparameters for mobile device 106 (see FIG. 1), which are specified bystructure 620.

FIG. 6C is an example of a structure 640 containing a specification ofapplication server log entry parameters utilized in the process of FIG.2, in accordance with embodiments of the present invention. In step 204(see FIG. 2), the selected log entries of application server 116 (seeFIG. 1) are specified by structure 640.

FIG. 7A is an example of a user interface 700, which presents faults andperformance anomalies detected in the process of FIG. 2, in accordancewith embodiments of the present invention. In repeated performances ofstep 208 (see FIG. 2), diagnostics and forensics engine 130 (see FIG. 1)detects multiple faults, performance anomalies, and other events ofinterest for mobile application 109 (see FIG. 1) over a specified periodof time. User interface 700 includes timestamps of requests sent to theserver from mobile device 106 (see FIG. 1) under the Request Timecolumn, an identifier of mobile device 106 (see FIG. 1) under the Devicecolumn, identifiers of servers under the Server column, RTT values or anindicator of a connection time out under the RTT column, response codesunder the Response column, a method identifier under the Method column,and hyperlink buttons labeled “Investigate” under the Investigatecolumn. In response to activating an Investigate button such as button716, a corresponding user interface is displayed that includesdiagnostic clues and a root cause analysis, which provides data for userto manually analyze the details of a fault to determine the likelylocation of a root cause of the fault.

User interface 700 includes RTT values 702, 704 and 706, which aredetermined by diagnostics and forensics engine 130 (see FIG. 1) toexceed a threshold value in step 208 (see FIG. 2). User interface 700also includes TIMEOUT indicators 708 and 710, which indicate failedconnections (i.e., the connection began but was never completed).Furthermore, user interface 700 includes response codes 712 and 714,which are HTTP response codes of 500 (i.e., a response code indicating aserver module failure).

FIG. 7B is an example of a user interface 750 that includes diagnosticclues and candidate root causes determined in the process of FIG. 3, inaccordance with embodiments of the present invention. User interface 750includes the details presented in response to a user activatingInvestigate button 716 (see FIG. 7A) in the fourth data row in userinterface 700 (see FIG. 7A). The details in user interface 750 allow auser to manually analyze the error response code 714 (see FIG. 7A) todetermine the likely root cause of the server module failure associatedwith response code 714 (see FIG. 7A).

User interface 750 includes timestamps and local times of log entriesfrom correlated logs that are relevant to the server module failureunder the Timestamp and Local Time columns, respectively. Under theSeverity column, user interface 750 includes a severity code of each logentry. For a severity code, I indicates that the entry providesinformation, R indicates that the entry describes an error (e.g., asystem error), W indicates that the entry describes a warning, and Oindicates that a level of severity has not been assigned to the entry.User interface 750 also includes identifiers of modules and componentsassociated with each log entry under the Module and Component columns,respectively. Furthermore, user interface 750 includes messages from thelog entries under the Message column, including messages 752 and 754.

The data rows in user interface 750 are the result of the temporalanalysis performed in step 310 (see FIG. 3). The textual analysisperformed in step 312 (see FIG. 3) detects the keyword of “exception” inmessage 752 and “failure” in message 754. In one embodiment, userinterface 750 highlights the messages 752 and 754, but not the othermessages, to indicate that keywords are detected in messages 752 and754. By focusing only on the highlighted messages, the user can quicklyanalyze the failure and determine the likely location of the root causeof the failure as identified in the corresponding entries under theModule and Component columns.

Computer System

FIG. 8 is a block diagram of computer 102 that is included in the systemof FIG. 1 and that implements the processes of FIG. 2, FIG. 3, and FIG.4, in accordance with embodiments of the present invention. Computer 102is a computer system that generally includes a central processing unit(CPU) 802, a memory 804, an input/output (I/O) interface 806, and a bus808. Further, computer 102 is coupled to I/O devices 810 and a computerdata storage unit 812. CPU 802 performs computation and controlfunctions of computer 102, including executing instructions included inprogram code 814 to perform a method of detecting and analyzing ananomaly in a performance of an application in a connection betweenclient and server computers, where the instructions are carried out byCPU 802 via memory 804. CPU 802 may include a single processing unit, orbe distributed across one or more processing units in one or morelocations (e.g., on a client and server).

Memory 804 includes a known computer readable storage medium, which isdescribed below. In one embodiment, cache memory elements of memory 804provide temporary storage of at least some program code (e.g., programcode 814) in order to reduce the number of times code must be retrievedfrom bulk storage while instructions of the program code are carriedout. Moreover, similar to CPU 802, memory 804 may reside at a singlephysical location, including one or more types of data storage, or bedistributed across a plurality of physical systems in various forms.Further, memory 804 can include data distributed across, for example, alocal area network (LAN) or a wide area network (WAN).

I/O interface 806 includes any system for exchanging information to orfrom an external source. I/O devices 810 include any known type ofexternal device, including a display device, keyboard, etc. Bus 808provides a communication link between each of the components in computer102, and may include any type of transmission link, includingelectrical, optical, wireless, etc.

I/O interface 806 also allows computer 102 to store information (e.g.,data or program instructions such as program code 814) on and retrievethe information from computer data storage unit 812 or another computerdata storage unit (not shown). Computer data storage unit 812 includes aknown computer-readable storage medium, which is described below. In oneembodiment, computer data storage unit 812 is a non-volatile datastorage device, such as a magnetic disk drive (i.e., hard disk drive) oran optical disc drive (e.g., a CD-ROM drive which receives a CD-ROMdisk).

Memory 804 and/or storage unit 812 may store computer program code 814that includes instructions that are executed by CPU 802 via memory 804to detect and analyze an anomaly in a performance of an application in aconnection between client and server computers. Although FIG. 8 depictsmemory 804 as including program code 814, the present inventioncontemplates embodiments in which memory 804 does not include all ofcode 814 simultaneously, but instead at one time includes only a portionof code 814.

Further, memory 804 may include an operating system (not shown) and mayinclude other systems not shown in FIG. 8.

Storage unit 812 and/or one or more other computer data storage units(not shown) that are coupled to computer 102 may store environmentattributes and performance data relative to the application session ofmobile application 109 (see FIG. 1), which are provided by device agent110 (see FIG. 1), network infrastructure agent 122 (see FIG. 1), andback-end infrastructure agent 120 (see FIG. 1).

As will be appreciated by one skilled in the art, in a first embodiment,the present invention may be a system; in a second embodiment, thepresent invention may be a method; and in a third embodiment, thepresent invention may be a computer program product.

Any of the components of an embodiment of the present invention can bedeployed, managed, serviced, etc. by a service provider that offers todeploy or integrate computing infrastructure with respect to detectingand analyzing an anomaly in a performance of an application in aconnection between client and server computers. Thus, an embodiment ofthe present invention discloses a process for supporting computerinfrastructure, where the process includes providing at least onesupport service for at least one of integrating, hosting, maintainingand deploying computer-readable code (e.g., program code 814) in acomputer system (e.g., computer 102) including one or more processors(e.g., CPU 802), wherein the processor(s) carry out instructionscontained in the code causing the computer system to detect and analyzean anomaly in a performance of an application in a connection betweenclient and server computers. Another embodiment discloses a process forsupporting computer infrastructure, where the process includesintegrating computer-readable program code into a computer systemincluding a processor. The step of integrating includes storing theprogram code in a computer-readable storage device of the computersystem through use of the processor. The program code, upon beingexecuted by the processor, implements a method of detecting andanalyzing an anomaly in a performance of an application in a connectionbetween client and server computers.

While it is understood that program code 814 for detecting and analyzingan anomaly in a performance of an application in a connection betweenclient and server computers may be deployed by manually loading directlyin client, server and proxy computers (not shown) via loading acomputer-readable storage medium (e.g., computer data storage unit 812),program code 814 may also be automatically or semi-automaticallydeployed into computer 102 by sending program code 814 to a centralserver or a group of central servers. Program code 814 is thendownloaded into client computers (e.g., computer 102) that will executeprogram code 814. Alternatively, program code 814 is sent directly tothe client computer via e-mail. Program code 814 is then either detachedto a directory on the client computer or loaded into a directory on theclient computer by a button on the e-mail that executes a program thatdetaches program code 814 into a directory. Another alternative is tosend program code 814 directly to a directory on the client computerhard drive. In a case in which there are proxy servers, the processselects the proxy server code, determines on which computers to placethe proxy servers' code, transmits the proxy server code, and theninstalls the proxy server code on the proxy computer. Program code 814is transmitted to the proxy server and then it is stored on the proxyserver.

Another embodiment of the invention provides a method that performs theprocess steps on a subscription, advertising and/or fee basis. That is,a service provider, such as a Solution Integrator, can offer to create,maintain, support, etc. a process of detecting and analyzing an anomalyin a performance of an application in a connection between client andserver computers. In this case, the service provider can create,maintain, support, etc. a computer infrastructure that performs theprocess steps for one or more customers. In return, the service providercan receive payment from the customer(s) under a subscription and/or feeagreement, and/or the service provider can receive payment from the saleof advertising content to one or more third parties.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) (memory 804 and computer data storageunit 812) having computer readable program instructions 814 thereon forcausing a processor (e.g., CPU 802) to carry out aspects of the presentinvention.

The computer readable storage medium can be a tangible device that canretain and store instructions (e.g., program code 814) for use by aninstruction execution device (e.g., computer 102). The computer readablestorage medium may be, for example, but is not limited to, an electronicstorage device, a magnetic storage device, an optical storage device, anelectromagnetic storage device, a semiconductor storage device, or anysuitable combination of the foregoing. A non-exhaustive list of morespecific examples of the computer readable storage medium includes thefollowing: a portable computer diskette, a hard disk, a random accessmemory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), a static random access memory(SRAM), a portable compact disc read-only memory (CD-ROM), a digitalversatile disk (DVD), a memory stick, a floppy disk, a mechanicallyencoded device such as punch-cards or raised structures in a groovehaving instructions recorded thereon, and any suitable combination ofthe foregoing. A computer readable storage medium, as used herein, isnot to be construed as being transitory signals per se, such as radiowaves or other freely propagating electromagnetic waves, electromagneticwaves propagating through a waveguide or other transmission media (e.g.,light pulses passing through a fiber-optic cable), or electrical signalstransmitted through a wire.

Computer readable program instructions (e.g., program code 814)described herein can be downloaded to respective computing/processingdevices (e.g., computer 102) from a computer readable storage medium orto an external computer or external storage device (e.g., computer datastorage unit 812) via a network (not shown), for example, the Internet,a local area network, a wide area network and/or a wireless network. Thenetwork may comprise copper transmission cables, optical transmissionfibers, wireless transmission, routers, firewalls, switches, gatewaycomputers and/or edge servers. A network adapter card (not shown) ornetwork interface (not shown) in each computing/processing devicereceives computer readable program instructions from the network andforwards the computer readable program instructions for storage in acomputer readable storage medium within the respectivecomputing/processing device.

Computer readable program instructions (e.g., program code 814) forcarrying out operations of the present invention may be assemblerinstructions, instruction-set-architecture (ISA) instructions, machineinstructions, machine dependent instructions, microcode, firmwareinstructions, state-setting data, or either source code or object codewritten in any combination of one or more programming languages,including an object oriented programming language such as Smalltalk, C++or the like, and conventional procedural programming languages, such asthe “C” programming language or similar programming languages. Thecomputer readable program instructions may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider). In some embodiments, electronic circuitry including, forexample, programmable logic circuitry, field-programmable gate arrays(FPGA), or programmable logic arrays (PLA) may execute the computerreadable program instructions by utilizing state information of thecomputer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations (e.g., FIG. 2, FIG. 3, and FIG. 4) and/or blockdiagrams (e.g., FIG. 1 and FIG. 8) of methods, apparatus (systems), andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerreadable program instructions (e.g., program code 814).

These computer readable program instructions may be provided to aprocessor (e.g., CPU 802) of a general purpose computer, special purposecomputer, or other programmable data processing apparatus (e.g.,computer 102) to produce a machine, such that the instructions, whichexecute via the processor of the computer or other programmable dataprocessing apparatus, create means for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks. Thesecomputer readable program instructions may also be stored in a computerreadable storage medium (e.g., computer data storage unit 812) that candirect a computer, a programmable data processing apparatus, and/orother devices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions (e.g., program code 814) mayalso be loaded onto a computer (e.g. computer 102), other programmabledata processing apparatus, or other device to cause a series ofoperational steps to be performed on the computer, other programmableapparatus or other device to produce a computer implemented process,such that the instructions which execute on the computer, otherprogrammable apparatus, or other device implement the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While embodiments of the present invention have been described hereinfor purposes of illustration, many modifications and changes will becomeapparent to those skilled in the art. Accordingly, the appended claimsare intended to encompass all such modifications and changes as fallwithin the true spirit and scope of this invention.

What is claimed is:
 1. A method of detecting and analyzing an anomaly ina performance of an application in a connection between client andserver computers, the method comprising: a first computer determining astatus code of a response from the server computer and determining thatthe status code is a Hypertext Transfer Protocol (HTTP) status code of500 through 599, which indicates the server computer did not properlyperform a function in response to a request from the client computerexecuting the application, the response being sent by the servercomputer to the client computer via a communications network andresponsive to the request; the first computer determining that theconnection timed out in response to the server computer not respondingto the request within a predetermined time period; the first computercalculating values of a round trip latency time (RTT) for multipleclient computers having application sessions with the server computer,the values of the RTT including a value of a RTT of the response; thefirst computer dividing a space of the values of the RTT into buckets ofRTT values, the buckets having a fixed size; the first computercomputing running counts and means for the values of the RTT in eachbucket; the first computer maintaining a boundary value that determineswhich buckets are in a lower value cluster C₁ employed by a k-meansclustering algorithm and which other buckets are in a higher valuecluster C₂ employed by the k-means clustering algorithm, wherein k=2;the first computer determining the buckets whose RTT values includerespective values of the RTT, assigning the values of the RTT to therespective buckets, re-computing the counts and means for each bucket,and balancing C₁ and C₂ to ensure that (i) values in C₁ are closer to amean μ₁ of C₁ and (ii) values in C₂ are closer to a mean μ₂ of C₂; thefirst computer computing μ₁ of C₁, a standard deviation σ₁ of C₁, μ₂ ofC₂, and a standard deviation σ₂ of C₂; the first computer computing athreshold value as μ₂₊2σ₂ if μ₁+σ₁≥μ₂ or as μ₁+σ₁ if μ₁+σ₁<μ₂; the firstcomputer determining that the value of the RTT of the response exceedsthe threshold value; based on the status code of the response being theHTTP status code of 500 through 599, the value of the RTT exceeding thethreshold value, and the connection having timed out in response to theserver computer not responding to the request within the predeterminedtime period, the first computer detecting the anomaly in the performanceof the application; and based on a temporal analysis and textualanalysis of log entries associated with the anomaly, and based on anenvironment analysis that determines activity of the client computer,the server computer, and the communications network, the first computerdetermining candidate root causes of a failure that resulted in theanomaly, the failure being in the client computer, the server computer,the communications network, or a combination of the client computer, theserver computer, and the communications network; the first computersending an alert to a user via a graphical user interface dashboard, thealert describing the anomaly; the first computer collecting attributesof the anomaly and sending the attributes to a machine learning process,the attributes including the RTT, the indication of whether theconnection timed out a delay value of the connection, details of theserver computer and the application, details about a function specifiedby the request, and a uniform resource locator of the server computer;in response to the alert, the first computer receiving feedback from theuser about whether the anomaly was correctly detected or incorrectlydetected; the first computer utilizing the feedback as a label of themachine learning process; based on the collected attributes, thefeedback, and the label, the first computer training a machine learningmodel for the machine learning process, the machine learning modelincluding rules specifying subsequent anomalies in the performance ofthe application; the first computer detecting a subsequent anomaly inthe performance of the application, the subsequent anomaly resulting inan outage in a network link, router, or switch in the communicationsnetwork, which results in a disconnection between the client computerand the server computer; the first computer determining that thesubsequent anomaly is correctly detected based on the trained machinelearning model; and based on the subsequent anomaly being determined tobe correctly detected based on the trained machine learning model, thefirst computer repairing the outage in the network link, the router, orthe switch, which reconnects the client computer and the server computervia the communications network.
 2. The method of claim 1, furthercomprising: the first computer determining a period of time relevant tothe anomaly; based on the period of time, the first computer selectingrelevant entities from among the client computer, the server computer,and components of the communications network; based on the selectedrelevant entities and the period of time, the first computer selectinglog entries from logs provided by the relevant entities; subsequent tothe selecting the log entries, the first computer filtering the selectedlog entries based on keywords that specify anomalies; the first computerdetermining a usage of a central processing unit (CPU) of the servercomputer, a usage of a memory by the server computer, and aninput/output (I/O) activity of the server computer; and based on thefiltered log entries, the usage of the CPU, the usage of the memory, andthe I/O activity, the first computer determining whether each of theclient computer, the server computer, and the components of thecommunications network was active or inactive at a time of an occurrenceof the anomaly, wherein the determining the candidate root causes isbased in part on whether each of the client computer, the servercomputer and the components of the communications network is determinedto have been active or inactive at the time of the occurrence of theanomaly.
 3. The method of claim 2, further comprising: the firstcomputer determining one or more components of the server computer wereactive at the time of the occurrence of the anomaly; and based on thefiltered log entries, the usage of the CPU, the usage of the memory, andthe I/O activity, the first computer determining whether the one or morecomponents of the server computer were performing tasks relevant to theapplication or extraneous to the application, wherein the determiningthe candidate root causes is based in part on whether the one or morecomponents of the server computer were performing tasks relevant to theapplication or extraneous to the application.
 4. The method of claim 1,further comprising: the first computer determining confidences of therespective candidate root causes, each confidence indicating how likelythe respective root cause is an actual root cause of the anomaly; andthe first computer presenting the candidate root causes in an orderwhich is based on the confidences.
 5. The method of claim 1, furthercomprising: the first computer determining the anomaly specifies a typeof the alert; the first computer determining a role of a user; the firstcomputer determining an association between the type of the alert andthe role of the user; and based on the association between the type ofthe alert and the role of the user, the first computer presenting thealert to the user, the alert notifying the user about the anomaly. 6.The method of claim 1, further comprising the step of: providing atleast one support service for at least one of creating, integrating,hosting, maintaining, and deploying computer-readable program code inthe computer, the program code being executed by a processor of thecomputer to implement the determining the status code of the response,determining that the connection timed out in response to the servercomputer not responding to the request, calculating the values of theRTT, dividing the space of the values of RTT in the buckets, computingthe running counts and the means for the values of the RTT in eachbucket, maintaining the boundary value, determining the buckets whoseRTT values include the respective values of the RTT, assigning thevalues of the RTT to the respective buckets, re-computing the counts andthe means for each bucket, balancing C₁ and C₂, computing μ₁ of C₁, σ₁of C₁, μ₂ of C₂, and σ₂ of C₂, computing the threshold value ifμ₁+σ₁<μ₂, determining that the value of the RTT of the response exceedsthe threshold value, determining the candidate root causes of thefailure that resulted in the anomaly, sending the alert to the user viathe graphical user interface dashboard, collecting the attributes of theanomaly, sending the attributes to the machine learning process,receiving the feedback from the user about whether the anomaly wascorrectly detected or incorrectly detected, utilizing the feedback asthe label of the machine learning process, training the machine learningmodel for the machine learning process, detecting the subsequent anomalyin the performance of the application, determining that the subsequentanomaly is correctly detected based on the trained machine learningmodel, and repairing the outage in the network link, the router, or theswitch.
 7. A computer program product, comprising: a computer-readablestorage device; and a computer-readable program code stored in thecomputer-readable storage device, the computer-readable program codecontaining instructions that are executed by a central processing unit(CPU) of a computer system to implement a method of detecting andanalyzing an anomaly in a performance of an application in a connectionbetween client and server computers, the method comprising: the computersystem determining a status code of a response from the server computerand determining that the status code is a Hypertext Transfer Protocol(HTTP) status code of 500 through 599, which indicates the servercomputer did not properly perform a function in response to a requestfrom the client computer executing the application, the response beingsent by the server computer to the client computer via a communicationsnetwork and responsive to the request; the computer system determiningthat the connection timed out in response to the server computer notresponding to the request within a predetermined time period; thecomputer system calculating values of a round trip latency time (RTT)for multiple client computers having application sessions with theserver computer, the values of the RTT including a value of a RTT of theresponse; the computer system dividing a space of the values of the RTTinto buckets of RTT values, the buckets having a fixed size; thecomputer system computing running counts and means for the values of theRTT in each bucket; the computer system maintaining a boundary valuethat determines which buckets are in a lower value cluster C₁ employedby a k-means clustering algorithm and which other buckets are in ahigher value cluster C₂ employed by the k-means clustering algorithm,wherein k=2; the computer system determining the buckets whose RTTvalues include respective values of the RTT, assigning the values of theRTT to the respective buckets, re-computing the counts and means foreach bucket, and balancing C₁ and C₂ to ensure that (i) values in C₁ arecloser to a mean μ₁ of C₁ and (ii) values in C₂ are closer to a mean μ₂of C₂; the computer system computing μ₁ of C₁, a standard deviation σ₁of C₁, μ₂ of C₂, and a standard deviation σ₂ of C₂; the computer systemcomputing a threshold value as μ₂₊2σ₂ if μ₁+σ₁≥μ₂ or as μ₁₊2σ₁ ifμ₁+σ₁<μ₂; the computer system determining that the value of the RTT ofthe response exceeds the threshold value; based on the status code ofthe response being the HTTP status code of 500 through 599, the value ofthe RTT exceeding the threshold value, and the connection having timedout in response to the server computer not responding to the requestwithin the predetermined time period, the computer system detecting theanomaly in the performance of the application; and based on a temporalanalysis and textual analysis of log entries associated with theanomaly, and based on an environment analysis that determines activityof the client computer, the server computer, and the communicationsnetwork, the computer system determining candidate root causes of afailure that resulted in the anomaly, the failure being in the clientcomputer, the server computer, the communications network, or acombination of the client computer, the server computer, and thecommunications network; the computer system sending an alert to a uservia a graphical user interface dashboard, the alert describing theanomaly; the computer system collecting attributes of the anomaly andsending the attributes to a machine learning process, the attributesincluding the RTT, the indication of whether the connection timed out adelay value of the connection, details of the server computer and theapplication, details about a function specified by the request, and auniform resource locator of the server computer; in response to thealert, the computer system receiving feedback from the user aboutwhether the anomaly was correctly detected or incorrectly detected; thefirst computer utilizing the feedback as a label of the machine learningprocess; based on the collected attributes, the feedback, and the label,the computer system training a machine learning model for the machinelearning process, the machine learning model including rules specifyingsubsequent anomalies in the performance of the application; the computersystem detecting a subsequent anomaly in the performance of theapplication, the subsequent anomaly resulting in an outage in a networklink, router, or switch in the communications network, which results ina disconnection between the client computer and the server computer; thecomputer system determining that the subsequent anomaly is correctlydetected based on the trained machine learning model; and based on thesubsequent anomaly being determined to be correctly detected based onthe trained machine learning model, the first computer repairing theoutage in the network link, the router, or the switch, which reconnectsthe client computer and the server computer via the communicationsnetwork.
 8. The computer program product of claim 7, wherein the methodfurther comprises: the computer system determining a period of timerelevant to the anomaly; based on the period of time, the computersystem selecting relevant entities from among the client computer, theserver computer, and components of the communications network; based onthe selected relevant entities and the period of time, the computersystem selecting log entries from logs provided by the relevantentities; subsequent to the selecting the log entries, the computersystem filtering the selected log entries based on keywords that specifyanomalies; the computer system determining a usage of a centralprocessing unit (CPU) of the server computer, a usage of a memory by theserver computer, and an input/output (I/O) activity of the servercomputer; and based on the filtered log entries, the usage of the CPU,the usage of the memory, and the I/O activity, the computer systemdetermining whether each of the client computer, the server computer,and the components of the communications network was active or inactiveat a time of an occurrence of the anomaly, wherein the determining thecandidate root causes is based in part on whether each of the clientcomputer, the server computer and the components of the communicationsnetwork is determined to have been active or inactive at the time of theoccurrence of the anomaly.
 9. The computer program product of claim 8,wherein the method further comprises: the computer system determiningone or more components of the server computer were active at the time ofthe occurrence of the anomaly; and based on the filtered log entries,the usage of the CPU, the usage of the memory, and the I/O activity, thecomputer system determining whether the one or more components of theserver computer were performing tasks relevant to the application orextraneous to the application, wherein the determining the candidateroot causes is based in part on whether the one or more components ofthe server computer were performing tasks relevant to the application orextraneous to the application.
 10. The computer program product of claim7, wherein the method further comprises: the computer system determiningconfidences of the respective candidate root causes, each confidenceindicating how likely the respective root cause is an actual root causeof the anomaly; and the computer system presenting the candidate rootcauses in an order which is based on the confidences.
 11. The computerprogram product of claim 7, wherein the method further comprises: thecomputer system determining the anomaly specifies a type of the alert;the computer system determining a role of a user; the computer systemdetermining an association between the type of the alert and the role ofthe user; and based on the association between the type of the alert andthe role of the user, the computer system presenting the alert to theuser, the alert notifying the user about the anomaly.
 12. The computerprogram product of claim 11, wherein the method further comprises: thecomputer system collecting attributes of the anomaly and sending theattributes to a machine learning process, the attributes including theRTT, the indication of whether the connection timed out; a delay valueof the connection, details of the server computer and the application,details about a function specified by the request, and a uniformresource locator of the server computer; the computer system receivingfeedback from the user about whether the anomaly was correctly detectedor incorrectly detected; the computer system utilizing the feedback as alabel of the machine learning process; based on the collectedattributes, the computer system generating a machine learning model forthe machine learning process, the machine learning model including rulesspecifying subsequent anomalies; the computer system updating themachine learning model continuously or at specified time intervals; andbased on the machine learning model or the updated machine learningmodel, the computer system detecting a subsequent anomaly in theperformance of the application, wherein the subsequent anomaly is morelikely to be accurately detected than the anomaly detected by the priorstep of detecting the anomaly.
 13. A computer system comprising: acentral processing unit (CPU); a memory coupled to the CPU; and acomputer readable storage device coupled to the CPU, the storage devicecontaining instructions that are executed by the CPU via the memory toimplement a method of detecting and analyzing an anomaly in aperformance of an application in a connection between client and servercomputers, the method comprising: the computer system determining astatus code of a response from the server computer and determining thatthe status code is a Hypertext Transfer Protocol (HTTP) status code of500 through 599, which indicates the server computer did not properlyperform a function in response to a request from the client computerexecuting the application, the response being sent by the servercomputer to the client computer via a communications network andresponsive to the request; the computer system determining that theconnection timed out in response to the server computer not respondingto the request within a predetermined time period; the computer systemcalculating values of a round trip latency time (RTT) for multipleclient computers having application sessions with the server computer,the values of the RTT including a value of a RTT of the response; thecomputer system dividing a space of the values of the RTT into bucketsof RTT values, the buckets having a fixed size; the computer systemcomputing running counts and means for the values of the RTT in eachbucket; the computer system maintaining a boundary value that determineswhich buckets are in a lower value cluster C₁ employed by a k-meansclustering algorithm and which other buckets are in a higher valuecluster C₂ employed by the k-means clustering algorithm, wherein k=2;the computer system determining the buckets whose RTT values includerespective values of the RTT, assigning the values of the RTT to therespective buckets, re-computing the counts and means for each bucket,and balancing C₁ and C₂ to ensure that (i) values in C₁ are closer to amean μ₁ of C₁ and (ii) values in C₂ are closer to a mean μ₂ of C₂; thecomputer system computing μ₁ of C₁, a standard deviation σ₁ of C₁, μ₂ ofC₂, and a standard deviation σ₂ of C₂; the computer system computing athreshold value as μ₂₊2σ₂ if μ₁+σ₁≥μ₂ or as μ₁₊2σ₁ if μ₁+σ₁<μ₂; thecomputer system determining that the value of the RTT of the responseexceeds the threshold value; based on the status code of the responsebeing the HTTP status code of 500 through 599, the value of the RTTexceeding the threshold value, and the connection having timed out inresponse to the server computer not responding to the request within thepredetermined time period, the computer system detecting the anomaly inthe performance of the application; and based on a temporal analysis andtextual analysis of log entries associated with the anomaly, and basedon an environment analysis that determines activity of the clientcomputer, the server computer, and the communications network, thecomputer system determining candidate root causes of a failure thatresulted in the anomaly, the failure being in the client computer, theserver computer, the communications network, or a combination of theclient computer, the server computer, and the communications network;the computer system sending an alert to a user via a graphical userinterface dashboard, the alert describing the anomaly; the computersystem collecting attributes of the anomaly and sending the attributesto a machine learning process, the attributes including the RTT, theindication of whether the connection timed out a delay value of theconnection, details of the server computer and the application, detailsabout a function specified by the request, and a uniform resourcelocator of the server computer; in response to the alert, the computersystem receiving feedback from the user about whether the anomaly wascorrectly detected or incorrectly detected; the first computer utilizingthe feedback as a label of the machine learning process; based on thecollected attributes, the feedback, and the label, the computer systemtraining a machine learning model for the machine learning process, themachine learning model including rules specifying subsequent anomaliesin the performance of the application; the computer system detecting asubsequent anomaly in the performance of the application, the subsequentanomaly resulting in an outage in a network link, router, or switch inthe communications network, which results in a disconnection between theclient computer and the server computer; the computer system determiningthat the subsequent anomaly is correctly detected based on the trainedmachine learning model; and based on the subsequent anomaly beingdetermined to be correctly detected based on the trained machinelearning model, the first computer repairing the outage in the networklink, the router, or the switch, which reconnects the client computerand the server computer via the communications network.
 14. The computersystem of claim 13, wherein the method further comprises: the computersystem determining a period of time relevant to the anomaly; based onthe period of time, the computer system selecting relevant entities fromamong the client computer, the server computer, and components of thecommunications network; based on the selected relevant entities and theperiod of time, the computer system selecting log entries from logsprovided by the relevant entities; subsequent to the selecting the logentries, the computer system filtering the selected log entries based onkeywords that specify anomalies; the computer system determining a usageof a central processing unit (CPU) of the server computer, a usage of amemory by the server computer, and an input/output (I/O) activity of theserver computer; and based on the filtered log entries, the usage of theCPU, the usage of the memory, and the I/O activity, the computer systemdetermining whether each of the client computer, the server computer,and the components of the communications network was active or inactiveat a time of an occurrence of the anomaly, wherein the determining thecandidate root causes is based in part on whether each of the clientcomputer, the server computer and the components of the communicationsnetwork is determined to have been active or inactive at the time of theoccurrence of the anomaly.
 15. The computer system of claim 14, whereinthe method further comprises: the computer system determining one ormore components of the server computer were active at the time of theoccurrence of the anomaly; and based on the filtered log entries, theusage of the CPU, the usage of the memory, and the I/O activity, thecomputer system determining whether the one or more components of theserver computer were performing tasks relevant to the application orextraneous to the application, wherein the determining the candidateroot causes is based in part on whether the one or more components ofthe server computer were performing tasks relevant to the application orextraneous to the application.
 16. The computer system of claim 13,wherein the method further comprises: the computer system determiningconfidences of the respective candidate root causes, each confidenceindicating how likely the respective root cause is an actual root causeof the anomaly; and the computer system presenting the candidate rootcauses in an order which is based on the confidences.
 17. The computersystem of claim 13, wherein the method further comprises: the computersystem determining the anomaly specifies a type of the alert; thecomputer system determining a role of a user; the computer systemdetermining an association between the type of the alert and the role ofthe user; and based on the association between the type of the alert andthe role of the user, the computer system presenting the alert to theuser, the alert notifying the user about the anomaly.
 18. The computersystem of claim 17, wherein the method further comprises: the computersystem collecting attributes of the anomaly and sending the attributesto a machine learning process, the attributes including the RTT, theindication of whether the connection timed out; a delay value of theconnection, details of the server computer and the application, detailsabout a function specified by the request, and a uniform resourcelocator of the server computer; the computer system receiving feedbackfrom the user about whether the anomaly was correctly detected orincorrectly detected; the computer system utilizing the feedback as alabel of the machine learning process; based on the collectedattributes, the computer system generating a machine learning model forthe machine learning process, the machine learning model including rulesspecifying subsequent anomalies; the computer system updating themachine learning model continuously or at specified time intervals; andbased on the machine learning model or the updated machine learningmodel, the computer system detecting a subsequent anomaly in theperformance of the application, wherein the subsequent anomaly is morelikely to be accurately detected than the anomaly detected by the priorstep of detecting the anomaly.